{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Surrogate gradient\n",
    "\n",
    "[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/brainpy/brainpy/blob/master/docs_version2/tutorial_toolbox/surrogate_gradient.ipynb)\n",
    "[![Open in Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/brainpy/brainpy/blob/master/docs_version2/tutorial_toolbox/surrogate_gradient.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "@[Sichao He](https://github.com/routhleck)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In recent years, spiking neural networks (SNNs) show their promising advantages in energy efficiency, fault tolerance, and biological plausibility. However, SNNs are difficult to train using standard gradient descent methods because their activation functions are discontinuous and have zero gradients almost everywhere. The commonly used way is to replace the non-differentiable spiking function with the surrogate gradient function. A surrogate gradient function is a smooth function that approximates the derivative of the activation function and allows gradient-based learning algorithms to be applied to SNNs.\n",
    "\n",
    "BrainPy provides multiple surrogate gradient functions with different properties of smoothness, boundedness, and biological plausibility. The full list is shown in Table below, and for the example of the surrogate gradient function please see the Figure Below. \n",
    "\n",
    "In practice, users can use these surrogate gradient functions as parameters in neuron models. For example, in the leaky integrate-and-fire (LIF) neuron model brainpy.neurons.LIF, use can use:\n",
    "\n",
    "``model = brainpy.neurons.LIF (... , spike_fun=<surrogate function >)``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../_static/surrogate_gradient.png\" width=\"700 px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Name                                   | Implementation                                 |\n",
    "|----------------------------------------|------------------------------------------------|\n",
    "| Sigmoid function                       | `brainpy.math.surrogate.sigmoid`               |\n",
    "| Piecewise quadratic function           | `brainpy.math.surrogate.piecewise_quadratic`   |\n",
    "| Piecewise exponential function         | `brainpy.math.surrogate.piecewise_exp`          |\n",
    "| Soft sign function                     | `brainpy.math.surrogate.soft_sign`             |\n",
    "| Arctan function                        | `brainpy.math.surrogate.arctan`                |\n",
    "| Nonzero sign log function              | `brainpy.math.surrogate.nonzero_sign_log`      |\n",
    "| Erf function                           | `brainpy.math.surrogate.erf`                   |\n",
    "| Piecewise leaky relu function          | `brainpy.math.surrogate.piecewise_leaky_relu`  |\n",
    "| Squarewave Fourier series              | `brainpy.math.surrogate.squarewave_fourier_series`|\n",
    "| S2NN surrogate spiking function        | `brainpy.math.surrogate.s2nn`                  |\n",
    "| q-PseudoSpike surrogate function       | `brainpy.math.surrogate.q_pseudo_spike`        |\n",
    "| Leaky ReLU function                    | `brainpy.math.surrogate.leaky_relu`            |\n",
    "| Log-tailed ReLU function               | `brainpy.math.surrogate.log_tailed_relu`       |\n",
    "| ReLU gradient function                 | `brainpy.math.surrogate.relu_grad`             |\n",
    "| Gaussian gradient function             | `brainpy.math.surrogate.gaussian_grad`         |\n",
    "| Multi-Gaussian gradient function       | `brainpy.math.surrogate.multi_gaussian_grad`   |\n",
    "| Inverse-square surrogate gradient      | `brainpy.math.surrogate.inv_square_grad`       |\n",
    "| Slayer surrogate gradient function     | `brainpy.math.surrogate.slayer_grad`           |\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
